PROBLEMS 

OF EDUCATION 

IN THE 21* CENTURY 
Volume 65, 2015 


18 





COMMUNICATION IN STEM EDUCATION: 
A NON-INTRUSIVE METHOD 

FOR ASSESSMENT & K20 

EDUCATOR FEEDBACK 


Mike Borowczak 
Erebus Labs, Laramie, Wyoming, USA 
E-mail: mike@erebuslabs.com 


Abstract 


The current push to educate highly qualified and productive Science, Technology, Engineering and Math- 
ematics (STEM) professionals typically culminates in collegiate capstone projects. These group projects 
attempt to prepare STEM professionals for entry into the real world as they exemplify early career group 
dynamics and expose the reality that communication and other soft-skills are often just as important as 
technical skills. Since attitudes of STEM students and early career professions are established throughout 
the entire K-20 curriculum, this article also provides some feedback to educators. The research presented 
utilized, a free cloud based collaboration tool to observe the communication habits of three senior cap- 
stone teams (n= 13 participants) and was able to predict communication success and failures through 
straight-forward analysis of several key parameters, including: discussion frequency, number of respons- 
es and the distribution of work over time. 

Key words: communication in STEM, communication assessment, senior capstones, non-intrusive soft- 
skill assessment, engineering education, ABET. 


Introduction 


A gross misconception exists amongst many K20 science, mathematics, engineering and 
technology (STEM) students — and the only way to eliminate it is by educating our students in 
a soft skill they are trying to avoid. Many future and current STEM students believe that their 
future career will only require a single technical skill set. Unfortunately, soft skills, such as ef- 
fective communication — are the cornerstone of every successful STEM practitioner, researcher, 
and educator. The only way to address this communication gap is by teaching our current K20 
students these soft-skills. Unfortunately, assessing student gains in soft-skill is a non-trivial 
task. There is no test for how well one communicates. The ability to communicate is generally 
only assessed through observation — a task that is not only tedious, but also biased and skewed 
when done in controlled environments (e.g., a classroom presentation). This paper focuses on 
an approach for unobtrusive observation of communication within teams and then demonstrates 
how objective measures can predict and align to observable communication successes and fail- 
ures. 


Problem of Research 


Communication is a critical component in creating well-rounded STEM citizens (Mel- 
lors-Bourne, Connor & Jackson, 2011; Carnevale, Smith, & Melton, 2011). Effective com- 
munication impacts careers, teams, businesses and collaborations and in a larger scope — how 
we as a people make public policy decisions. This work is grounded in three key aspects of 
communication and education; first and foremost, the importance of communication in STEM 
education; secondly, existing methods and practices for the integration of communication into 
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K-20 educational practices; and finally, the current state of assessing communication compe- 
tency in K-20. 


Importance of Communication 


The importance of communication within STEM education is steeped in existing litera- 
ture. Students who are ultimately successful are not only problem solvers with highly technical 
skill-sets, but also are effective at communication, collaboration, and understand the relation- 
ships between society and science (Brewer & Smith, 2011). The complexities of integrating 
STEM and communication, according to Davis, Beyerlein, and Davis (2005) are due to “mis- 
guided or mismatched understandings” of what high-quality performance actually means for 
practicing engineering professionals. Davis et al. (2006) highlight four areas that capture future 
STEM related professional performance; two of the areas deal with soft-skills, including com- 
munication, in both a personal capacity as well as in team-processes. These skills, they claim, 
“support technically sound and responsible design.” With team-oriented practices at the cor- 
nerstone of industry and academia, it’s critical to note that the top three common difficulties 
detected during team-based projects are manifestations of communication issues. These issues 
range from the obvious “poor communication among members” to higher-level applied com- 
munication issues of “poor leadership” and “failure to compromise” (Dutson, Todd, Magleby 
& Sorensen, 1997, p. 24). Communication, and other soft skills are so critical, that of the eleven 
student outcomes which the Accreditation Board for Engineering and Technology (ABET) re- 
quires engineering programs to show evidence of, only five relate to technical abilities, while 
the remaining six focus on soft skills (Williams, 2001). While highlighting the importance of 
communication grounds in this work, it is important to understand that K-20 education practic- 
es already incorporate a variety of communication centric approaches. Unfortunately, important 
workplace soft-skills such as teamwork and communication receive “relatively little attention 
from the engineering education research community” (Singer & Schweingruber 2012). 


Communication in K-20 


A variety of approaches, from specific methods to entire curriculums, already exist in 
K-20 classrooms to build and develop the communication skills of students. The newly cre- 
ated Next Generation Science Standards propose that science education requires students to 
develop explanations, evidence-based arguments, discuss open-ended questions, and engage in 
discussions (National Research Council, 2015). Several universities have created minors in En- 
gineering Communication and Performance in order to provide formal training in “performance 
skill’s necessary for success” with an objective of making their engineering graduates “effec- 
tive communicators” and “socially adept” (Seat, Parsons & Poppen, 2001, p. 7). While STEM 
lecture-based courses have been paired with laboratory style course, a recent shift in practice 
enhances laboratory experience with collaborative learning in order to engage student driven 
discussion throughout group work, short-term projects and full-term design projects (Gol & 
Nafalski, 2007). Irrespective of a specific course, curriculum or grade level, a common method 
of engaging students in non-rote exercises is through Problem Based Learning (PBL) activi- 
ties. Savage, Chen, and Vanasupa (2007) highlight that one of the key elements for successful 
PBL activities is that “team building exercises should be utilized to facilitate the development 
of trust and communication within the team” (p. 5). These activities are exemplified in culmi- 
nating, baccalaureate terminal courses, required by many STEM disciplines. These courses 
are often referred to as senior capstones, or senior design projects. A 2004 national study of 
STEM capstone courses found that 88% of respondents’ capstone projects were team-centric 
(McKenzie, 2004). With over two decades of communication focused curriculums and methods 
used in K-20 classrooms and still a perception that young STEM professionals are ill prepared 
to communicate — our assessment of STEM students’ communication skills must be addressed. 
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Assessment of Communication 


While assignments, courses, instructors, curriculums and accreditation boards have 
specific target outcomes and objectives related to communication, the question of how to ac- 
curately gather evidence and assess student gains remains loosely defined within the STEM 
disciplines. Gotel, Scharff and Seng (2006) describe a course in which communication was a 
critical component and those activities were “recorded using a weekly questionnaire” (p. 11) 
that was meant to examine details about communication participants, topic, frequency, timing 
and co-location. Their findings on communication focus on the models, mode, intensity and 
topics of communication, but when looking at the lessons for education and practice no clear 
assessment methods were identified. While literature on assessing STEM communication is 
limited, some work has begun in creating active listening skill assessments (Wilkins, Bern- 
stein, Harrison, Bekki, & Atkinson, 2011). Wilkins et al. developed a 42-item instrument to 
measure STEM student’s knowledge, ability and self-efficacy with respect to active listening. 
By their own admission, no formal observation of the student’s actual ability was measured 
since “actual observations can be costly and time intensive.” Assessing data collected from 
surveys, focus groups and student deliverables, a northwestern American university redesigned 
their capstone program with added emphasis on student learning and application of “project 
management, time management, teamwork, and communication skills” (Faust, Greenberg, & 
Pejcinovic, 2012, p. 1). The findings of Faust et al. (2012) also point to a disconnect between 
self and external assessments of team cohesion — with external assessors unable to detect the 
lower performing groups. 


Research Focus 


This research highlights a mechanism to capture and assess team communication dy- 
namics without relying directly on participant perceptions. Rather, this work uses statistics, 
trends, unprompted question and answer (Q&A) forums and documents captured through a 
free, online, distributed version control system. This approach provides not only a first pass at 
an objective and quantitative view of individual and team communication, but also the ability 
to use artifacts as qualitative support. 


Methodology of Research 
General Background of Research 


The research study presented here focuses on two types of data — quantitative data that 
was directly collected from an existing cloud-based collaborative version control tool as well 
as qualitative data stored within the same system. This mixed method study focused on three 
unique capstone projects occurring either during the 2013-2014 or 2014-2015 academic years. 
The capstone projects within this study were conducted at an American university with whom 
the researcher has no direct affiliation. The study participants consist of senior engineering 
students within the three separate teams. The study looked at the interactions between the par- 
ticipants themselves, an academic advisor assigned to each team, and an external sponsor for 
each team. The researcher used the following two questions in guiding the development and 
execution of the study: 


1. How can the communication of senior capstone teams be assessed? 
2. How can the communication of senior capstone teams inform K20 educators? 


Using these questions as a foundation, the researcher first examined communication 
frequency, engagement and depth through analysis of well-defined numerical metrics. The re- 
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searcher then coded communications and non-technical documents produced by the team to 
determine any underlying patterns. A follow-up survey was requested and completed by a sub- 
set of the participants (n=9). The researcher used the per-group completion percentage of the 
survey to validate any correlation between underlying communication patterns detected in the 
open response coding and the purely analytical processing of the numerical metrics. 


Sample of Research 


The participant sample in this study contains three groups of senior students studying 
Computer Science & Engineering at a single American university during the 2013/2014 and 
2014/2015 academic years that interacted with three academic instructors and two project spon- 
sors. The total participant sample of 17 consisted of the 12 undergraduate students, three in- 
structors, and the two project sponsors. Of the total participant sample, there were three females 
—one instructor, one student, and one sponsor. The three student groups were all assessed using 
the same methods and procedures after collecting data in an identical fashion during both aca- 
demic years. For simplicity and anonymity, the three groups will only be identified as groups 
Ada, Bohr and Curie; individual students within the groups will not be identified. 


Instrument and Procedures 


In order to reduce complexity of reuse by future researchers and instructors, a single 
online cloud-based tool was used to capture a majority of both the quantitative and qualitative 
data. Unlike traditional assessment instruments, the researcher re-purposed an existing online 
collaborative tool in order to transparently collect data without interfering with the workflow 
or habit of the senior capstone teams. For this particular study, the researcher used a version 
control and project management tool called git. Version control systems give users the abil- 
ity to save work and see exact changes over time, and if needed, revert parts and pieces or 
entire documents back to a previous version. Specially, the capstone sponsor requested that 
each capstone team used github, a cloud-based instance of the git tool that contains additional 
collaborative features. The additional features of github that were used included the ability to 
create a wiki — or an online journal, and “open issues” when question or problems were found. 
At the beginning of the capstone project, all student participants were asked to use the tool as a 
centralized location for all development work, deliverables, meeting notes and questions both 
internal to the group as well as externally to project sponsors. In order to remove any bias from 
the researcher and sponsor, data from the tool was only downloaded and analyzed after the con- 
clusion of the capstone project experience. In addition to the use of the tool, participants were 
also asked at both the beginning and end of the project to complete a 20-minute open response 
survey — the study presented here only uses data related to the response rate per team. 


Data Analysis 


The cloud based tool used in this work allows insight into several numerical parame- 
ters, key to communication, including, with respect to a particular individual: number of emails 
sent, number of discussions started, number of discussions responses, length of each discussion 
thread and workload trends (when work was added to the cloud based tool). Additionally, the 
researcher identified the distribution of committed work over time, as well as the use of other 
collaborative features within the tool. The researcher then collected information on the number 
of meetings that teams held — identifying if they were team member only meetings or with the 
sponsor as well. Additionally, the number of responses to a pre and post survey was collected 
and normalized over the number of participants in a given group. Finally, the participants’ 
weekly reports were coded for communication topics. In total, this research presents findings 
surrounding 11 quantifiable metrics. 
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Results of Research 


While a total of 11 quantifiable metrics were collected, four were based on the research- 
ers’ observations, while the remaining seven were collected through the cloud based collabo- 
ration tool. The four metrics collected “by-hand” are the ones that would typically be used to 
gauge communication and focus on meeting frequency and team engagement. The tool allowed 
the researcher to capture seven quantifiable metrics surrounding communication without any 
inference, bias or intrusion into the team dynamics. The captured metric ranged from the num- 
ber of discussions (email, online and total); the average length of each discussion — as measured 
by the number of responses; how work was distributed based on day of the week and time of 
day; what the work frequency of the group appeared to be over time; and finally, whether or not 
the team took advantage of collaborative features within the online tool. In order to quantify the 
work distribution and work overtime, the researcher used the charts shown in Figure | to clas- 
sify work distribution as either evenly spread over time (Ada and Curie) or unevenly (Bohr). 
More specifically, note that the Ada and Curie teams have an evenly distributed work schedule 
across all days afternoon and into early morning, while the Bohr team has a wider variance of 
submitted work. Figure 2 shows the charts used by the researcher to quantify the pattern of 
work as increasing (Ada and Curie) or decreasing (Bohr). 


Sunday e e ee e@ ee ee@ @ee e °@ ° 
Monday @ ° @e e @e e Monday e ° er e@@ e 
1 e °@@e« ee e Tuesday -°@ e@ee . 
dnesd e e eceeeeo e@o@ee Wednesd e ee occ e@> 


Thured e e©«@ ee e220 ®@ e Thundy @@@e . eeee ° ° 
Fidy  @@e e@e@@ee ee Friday ° ©e+7@@re-re 
Saturday e o e@@ 2 ce e Saturay @ © e 
2 aman as 100 20 © ep 1 a ma Se eT ba MMe HA NZD ND 2D > AD SDD TH BD BD 109 TD 
(a) Ada (b) Bohr 
Sunday *e@@-+>-@eee: 
e e ee@e@ree eee 
@e ee oe Or ees O@ee © O@e-@ 
Wednesday e ee . eee @eee @®@ @Oe BS 
Thusdey @ @ ee ° ° eoer@e eee@ee 
Fri ee °“@-e @ e 
Sature day @ee e@eee 
Oo ta 2 tok Bebe oo atta tp 7 op 
(c) Curie 


Figure 1: Comparison of work distribution for three teams. Each row is a day 
of the week; each column is an hour during the day. The size of each 
circle represents a normalized view of work submitted during the day 
and hour combination. 
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a) Ada b) Bohr c) Curie 


Figure 2: Comparison of pattern of work over time for three teams over a 
6-month period: (a) Ada, (b) Bohr and (c) Curie. Each tick mark 
represents the end of a month. 


Table 1 shows an aggregated view of the 11 quantifiable metrics for the three participant 
groups in this study. The first seven metrics (1-7) are directly taken from the online tool, while 
the last four (8-11) are derived from the researchers observation of group interactions. The 
11 quantifiable results mirror the data collected and include in order: 1) the number of email 
discussion of the group, 2) the number of online discussion carried out by the discussion, 3) 
the total number of discussions, 4) the length of the discussion (as measured by the number of 
replies to a discussion thread), 5) how evenly distributed was the work (based on classification 
of Figure | data), 6) the work frequency over time (based on classification of Figure 2 data), 7) 
whether or not teams used additional collaborative tools, 8) the number of team meeting with 
the sponsor, 9) the number of meeting without the sponsor, 10) the total number of meetings and 
finally, 11) the percentage of the team responding to a post-survey. 

Immediately the Bohr participant group stands out with low marks in the metrics derived 
from the online tool — low total number of sponsor visible discussions, uneven work frequency 
and lack of collaborative tool usage. When comparing these metrics to the other four quantita- 
tive metrics we easily see that this group also has low marks in meeting with each other and the 
sponsor — with the team also only having a 33% response percentage to the pre and post survey. 
While the two remaining groups are more evenly paired, it is interesting to note that the Curie 
team did not utilize the additional collaborative features supplied by the online tool— rather, 
they supplemented communication through the use of email and additional sponsor meetings. 
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Table 1. The eleven quantifiable metrics for three teams: bolded items indicate 
the highest unique value in a category while those italicized are the 
lowest unique value. 





Participant Teams 



































Topic Ada Bohr Curie 
Email Discussions 16 9 56 
Online Discussions 96 4 0 
Total Discussions* 112 13 56 
Toe Eeved Avg. Discussion Length 4.9 1.9 3.4 
Metrics 
Even Distribution of Work Yes No Yes 
Work Over Time Increasing Decreasing Increasing 
Use of Collaborative Features Yes No No 
Sponsor + Team Meeting 6 2 9 
Observed Team Meetings 20 8 20 
Metrics Total Meetings 26 10 29 
Responded to Survey 80% 33% 100% 





Themes in Artifacts 


When coding documents from Ada, a theme of expert pairs emerged where any issues 
were discussed in small collaborative pairings then brought to the larger group for suggestions 
and input. The team also regularly communicated what they had done, what they were planning 
on doing and what issues they currently had —with respect to the project as well as in the other 
academic and personal endeavors. The Bohr team also submitted weekly-based status reports, 
but they were less natural, and ended well before the conclusion of the term. From the early 
status reports, though past the midway of the term comments within the documents such as 
“teammate refused to ...” were prevalent across several team members. Additionally, meeting 
minutes pointed out advice from their academic advisors to “have regular [weekly] meetings.” 
The general theme that emerged from the Bohr documents was one of disconnected experts 
who had trouble establishing any regular pattern of communication. Finally, the Curie team 
created collaborative follow-up documents after each meeting — as a smaller sized team each 
member became a content expert, but determined and communicated as a team — their docu- 
ments were filled with “[member_X] has volunteered to take lead on ...” and “[member_Y] and 
[member _Z] have shared ... with the rest of the team.” 


Discussion 


Recall the two driving questions for this research. The first dealt with assessment of 
communication in capstone teams, while the second surrounded using capstone communication 
trends to inform K20 educators. In regards to the first question — when separating out the major 
themes from each group it was apparent that there were two groups, Ada and Curie, that were 
communicating and in sync, while the Bohr group failed to establish a routine that would allow 
them to communicate effectively. Using the last four observed metrics from Table 1, which fol- 
low typical communication observation metrics, a pattern of collaboration and communication 
was easily identified - with the Ada and Curie teams meeting two and half to almost three times 
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more than the Bohr team. Next, when analyzing the seven tool based metrics, the Ada group 
clearly stands out in the number of online discussion as well as the length of each discussion 
and the distribution of work. The longer discussion lengths of the Ada and Currie team implies 
non-trivial collaboration with deeper and richer conversations which is in contrast to the shorter 
length of discussion within the Bohr team — with most discussions being a single question as- 
sociated with a single response. Furthermore, the even distribution of when work occurred for 
the Ada and Curie teams suggests that both of these teams were able to align themselves on 
common work and collaboration schedules — a critical key for effective communication. Es- 
tablished within the research study were clear patterns of behavior present in easily quantifiable 
metrics of communication frequency, discussion length and distribution of work over time. 
Future work is needed in increasing the sample size of the study, defining the interpretations of 
the distribution of work and the work over time metrics as well as more in-depth coding and 
analysis of artifact themes. 

While a relationship exists between the seven tool-based metrics and the four observed 
metrics in their ability to measure communication in capstone teams, the feedback for K20 
educators is perhaps more subtle. While there are clear communication successes, it is currently 
more difficult to attribute success to specific K20 practices; as such the focus of this discussion 
is on the observable root cause to the communication issues. In this vein, the Bohr team had 
resource conflicts (of time) and identity conflicts (between team members), which impeded 
their ability to communicate — not only with themselves, but with their sponsor and academic 
advisor as well. K20 educators can enhance their group-based projects by injecting benign 
conflicts into their scenarios — giving students exposure to solving these types of issues early 
in their education. Furthermore, effective communication within a team does not immediately 
translate to effective communication outside of the team — K20 educator can promote internal 
team communication as well as team to “assessor” communication — mirroring real world in- 
stances in which a team is always judged by some outside or higher-level entity. Finally, a note 
about richness of communication — a key parameter of effective communication hinges on the 
ability to have continuous refinement during a Q&A session. The three groups in question had 
varying levels of success in this area — with Ada having Q&A discussions with an average of 
five interactions, while the Bohr team only had 1.9 interactions per session. Focus and attention 
to effective questioning (and active listening) are critical to enhancing team dynamics. 


Conclusions 


The goal of this study was to determine if the communication patterns and abilities 
detected in the weekly reports, survey completion and meeting frequency could be predicted 
from the data collected using a cloud based collaboration tool. When looking at the three senior 
capstone groups within the study, two groupings clearly emerged — two who communicated 
well and one that did not. The data presented supports the ability to detect communication abili- 
ties or issues through analysis of built in metrics for a common STEM based collaboration tool. 
While future research is needed, current K20 teachers can build upon the work here as a first 
pass to detecting communication issues within groups — whether it is an elementary classroom 
PBL project or a senior capstone — by looking for the length of conversations in terms of the 
number of interactions, team dynamics, and the pattern of when work is produced. Purely data 
driven decisions are not always possible — but for group-based communication, given the right 
tools, we can gain incredible, quantifying insights without having to rely on the perceptions of 
those we are trying to assess. 
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